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FOREWORD 

This  research  was  performed  under  engineering  development  subproject  Z1252-PN.01 
(Revised  Recruiting  Systems).  It  was  sponsored  by  the  Chief  of  Naval  Operations  Counter 
Attrition  Task  Force.  The  objective  of  the  subproject  was  to  develop  and  test  marketing 
strategies  for  recruiting  a  target  population  of  19-  to  23-year-old,  unmarried,  nonprior- 
service,  male  high  school  graduates. 

This  is  the  third  of  three  reports  that  describe  the  development  and  evaluation  of 
recruiting  techniques  designed  to  reach  the  19-  to  23-year-old  age  group.  The  first  report 
(NPRDC  SR  82-22)  identified  peer  networking  and  direct-mail  marketing  as  two  promising 
strategies.  The  second  report  (NPRDC  SR  83-11)  described  the  peer  networking  strategy 
based  on  expanded  use  of  the  Navy's  Recruiting  Assistance  Program.  This  report 
describes  the  test  and  evaluation  of  the  direct-mail  marketing  strategy  in  recruiting  19-to 
23-year-olds  into  the  Navy. 

Test  and  evaluation  of  the  direct-mail  strategy  were  conducted  under  contract 
N00123-79-C-1511  with  Westat,  Inc.  The  contract  officer’s  technical  representatives 
were  Raye  Newmen  and  Kathleen  Fernandes. 


JAMES  F.  KELLY,  JR. 
Commanding  Officer 


JAMES  W.  TWEEDDALE 
Technical  Director 
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SUMMARY 


Problem 


The  Navy  recruiting  environment  in  the  1980s  is  one  in  which  the  supply  of  17-  and 
18-year-olds  is  steadily  declining,  while  the  number  of  ships  that  must  be  manned  is 
increasing.  One  way  to  expand  the  supply  of  recruits  is  to  direct  recruiting  efforts  at  an 
older  market  consisting  of  19-  to  23-year-olds.  To  take  advantage  of  this  source  of 
manpower,  recruiting  techniques  effective  in  enlisting  this  segment  of  the  population  need 
to  be  explored. 

Objective 

The  objective  of  this  effort  was  to  evaluate  the  effectiveness  of  a  tailored  direct- 
mail  marketing  strategy  for  recruiting  19-  to  23-year-old,  unmarried,  nonprior-service 
males  who  have  received  high  school  diplomas.  This  effort  was  not  designed  to  evaluate 
direct-mail  marketing  in  general,  but  rather  to  determine  the  extent  to  which  the 
strategy  could  be  effective  in  appealing  to  a  somewhat  older  audience. 

Approach 

A  direct-mail  campaign  was  designed  in  which  mailing  lists  and  campaign  materials 
were  directed  towards  the  interests  of  the  19-  to  23-year-old  age  group.  Mailing  lists 
were  obtained  from  automobile  and  motorcycle  magazines  with  a  high  proportion  of 
readership  in  this  age  range.  Tailored  direct-mail  materials  were  developed  and  tested 
for  their  appeal  to  target  group  members.  The  materials  were  distributed  in  February  and 
May  1981  to  about  32,000  magazine  subscribers  in  the  recruiting  territory  surrounding 
selected  recruiting  stations  in  Navy  Recruiting  Districts  (NRDs)  San  Francisco  and 
Seattle.  Subscribers  interested  in  learning  more  about  the  Navy  mailed  back  a  reply  card 
that  was  checked  for  age  eligibility  and  forwarded  to  the  appropriate  recruiting  station 
for  follow-up.  The  effectiveness  of  the  direct-mail  strategy  was  determined  by 
comparing  the  numbers  of  enlistments  produced  in  this  group  of  subscribers  with  the 
number  produced  in  a  group  who  did  not  receive  the  materials. 

Results 


Of  the  367  reply  cards  received,  177  were  forwarded  as  leads  for  recruiter  follow-up. 
Only  24  percent  of  the  leads  were  in  the  19-  to  23-year-old  age  group.  This  figure  was 
less  than  the  percentage  of  their  subscribers  that  the  magazines  estimated  to  be  in  the 
target  group  and  less  than  the  percentage  of  enlistees  in  the  two  NRDs  during  the  field 
test  period  who  were  19-  to  23-years  old.  The  conversion  rate  of  eligible  leads  enlisting  in 
the  Navy  during  the  field  test  period  was  lower  than  but  not  significantly  different  from 
the  conversion  rate  for  national  leads  in  Navy  Recruiting  Area  Eight  during  FY81.  In 
addition,  the  enlistment  rate  for  subscribers  who  participated  in  the  direct-mail  campaign 
was  lower  than  but  not  significantly  different  from  the  rate  for  subscribers  who  did  not 
receive  the  materials. 

Discussion 


The  following  factors  should  be  considered  in  interpreting  the  lack  of  impact  of  the 
direct-mail  campaign  on  enlistments: 

1.  The  size  of  the  direct-mail  campaign  and/or  the  number  of  magazine  subscribers 
in  the  target  population  may  have  been  insufficient  to  provide  a  fair  test  of  the  strategy. 


vU 


2.  The  manner  in  which  the  evaluation  of  the  campaign  was  constructed  may  have 
been  inappropriate  for  measuring  its  impact  on  enlistments. 

3.  The  content  of  the  direct-mail  letter  may  not  have  appealed  to  members  of  the 
target  population. 

4.  Given  the  positive  recruiting  environment  during  the  field  test  period,  recruiters 
may  have  believed  that  direct-mail  leads  were  likely  to  produce  enlistments  and  so 
assigned  them  a  low  priority  for  follow-up. 

Although  the  direct-mail  strategy  was  not  effective  in  enlisting  19-  to  23-year-olds, 
the  procedures  used  to  calculate  enlistment  and  conversion  rates  may  be  applicable  to 
other  direct-mail  evaluations.  A  comparison  of  these  measures  with  those  used  by  the 
Navy  Recruiting  Command  could  provide  information  on  the  validity  of  both  measurement 
techniques  that  might  lead  to  enhancement  of  the  methods  used  to  evaluate  direct-mail 
efforts. 

Conclusions 

The  direct-mail  campaign  did  not  have  a  significant  impact  upon  enlistment  rates  and 
was  not  effective  in  appealing  to  the  target  population  of  19-  to  23-year-olds. 
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INTRODUCTION 


Problem 

In  today's  recruiting  environment,  the  pool  of  high-school  age  youth  is  beginning  to 
decline.  At  the  same  time,  the  Navy  is  facing  increased  manpower  demands  under  plans 
to  expand  the  size  of  the  fleet.  One  method  of  expanding  the  pool  of  potential  Navy 
recruits  would  be  to  direct  recruiting  efforts  at  the  19-  to  23-year-old  age  group. 
Through  most  of  the  1980s,  the  size  of  this  group  will  be  expanding  as  the  large  pool  of 
high-school  age  youth  that  existed  in  the  1970s  becomes  older.  Given  the  potential 
importance  of  this  somewhat  older  age  group  to  recruiting,  techniques  need  to  be 
developed  to  enlist  a  larger  proportion  of  this  population  in  the  Navy.1 


Objective 


The  objective  of  this  effort  was  to  evaluate  the  effectiveness  of  a  tailored  direct- 
mail  marketing  strategy  for  recruiting  19-  to  23-year-old,  unmarried,  nonprior -service 
males  who  have  received  high  school  diplomas.  This  effort  was  not  designed  to  evaluate 
direct-mail  marketing  in  general,  but  rather  to  determine  the  extent  to  which  the 
strategy  could  be  effective  in  appealing  to  a  somewhat  older  audience. 


Background 


During  an  earlier  phase  of  the  current  effort,2  direct-mail  marketing  was  identified 
as  a  strategy  that  had  promise  for  attracting  members  of  the  target  group  into  the  Navy 
and  should  be  tested.  In  that  phase,  it  was  concluded  that  the  direct-mail  campaign 
should  be  targeted  to  the  19-  to  23-year-old  age  group  by  using  mailing  lists  and  campaign 
materials  directed  towards  the  interests  of  this  group.  Because  19-  to  23-year-old  males 
are  very  interested  in  all  sorts  of  motor  vehicles  and  subscribe  to  vehicle  specialty 
magazines,  it  was  recommended  that  the  mailing  lists  be  selected  from  those  automobile 
and  motorcycle  magazines  with  a  high  proportion  of  readership  in  this  target  group.  In 
addition,  it  was  recommended  that  the  materials  used  in  the  direct-mail  campaign  be 
tailored  to  the  target  population  by  testing  their  appeal  to  members  of  the  19-  to  23-year- 
old  age  group. 


APPROACH 


Experimental  Design 


Sixty-seven  recruiting  stations,  50  in  the  California  portion  of  Navy  Recruiting 
District  (NRD)  San  Francisco  and  17  in  the  Washington  portion  of  NRD  Seattle,  were 
randomly  assigned  to  experimental  or  control  groups  for  the  test.  Minor  adjustments 
were  made  in  the  assignments  to  accommodate  NRD  consolidation  of  several  recruiting 


lThese  techniques  focus  on  accessions  acquired  through  the  standard  recruitment 
process  and  do  not  consider  accessions  generated  through  lateral  entry  programs. 

2RomancziJ<,  A.  P.,  Goodstadt,  B.  E.,  Colby,  C.  L.,  &  Fernandes,  K.  Identification  of 
strategies  for  penetrating  the  19-  to  23-year-old  recruiting  market  (NPRDt  SR  82-22). 
San  Diego:  Navy  Personnel  Research  and  Development  Center,  April  1982.  (AD-A114 
261) 


1 


stations  and  to  match  experimental  and  control  stations  on  average  recruiter  produc¬ 
tivity. 


Magazine  subscription  lists  were  obtained  for  the  zip  codes  associated  with  the 
recruiting  territory  surrounding  the  34  experimental  and  33  control  stations.  The  names 
on  each  mailing  list  were  identified  as  belonging  to  the  experimental  or  control  group  on 
the  basis  of  the  subscriber's  zip  code.  The  direct-mail  materials  were  distributed  to 
subscribers  in  the  experimental  group.  The  effectiveness  of  the  direct-mail  strategy  was 
determined  by  comparing  accessions  produced  in  this  group  with  accessions  produced  in 
the  control  group  of  subscribers  who  did  not  receive  the  materials.3 

Development  of  the  Direct-mail  Letter 

Six  direct-mail  letters  were  tested  in  San  Francisco  and  Seattle  on  36  civilians  having 
the  characteristics  of  the  target  population  (i.e.,  19-  to  23-year-old,  unmarried  males  with 
high  school  diplomas,  of  various  racial/ethnic  backgrounds,  who  were  in,  or  trying  to 
enter,  the  labor  force).  The  testing  was  done  in  June  1980  during  focus  group  sessions 
that  were  conducted  to  determine  how  the  19-  to  23-year-old  age  group  was  affected  by 
current  recruiting  practices  and  appeals.  Participants  rated  each  letter  on  a  variety  of 
dimensions  (e.g.,  convincing,  easy  to  read)  and  selected  the  letter  that  would  most  make 
them  want  to  "find  out  more  about  the  Navy"  (see  footnote  2). 

Based  on  participants'  ratings,  it  was  determined  that  two  letters  had  the  strongest 
potential  for  inducing  information-seeking  behavior;  one  emphasized  the  Navy  as  a 
"school"  providing  training  and  experience;  and  the  other,  world  crises,  America's 
weakening  defense,  and  the  individual  taking  of  responsibility.  The  Commander,  Navy 
Recruiting  Command  (COMNAVCRUITCOM)  recommended  against  using  the  "crisis" 
letter  because  of  its  politically  sensitive  content.  Therefore,  following  minor  revision  by 
COMNAVCRUITCOM,  the  "school"  letter  was  approved  for  the  test,  printed  on  Navy 
letterhead,  and  signed  by  a  member  of  the  COMNAVCRUITCOM  staff.  A  copy  of  the 
letter  is  included  in  the  appendix. 

Selection  and  Adjustments  of  Mailing  Lists 

Demographic  information  on  the  readership  of  approximately  50  car  and  motorcycle 
publications  was  gathered  to  identify  those  with  the  highest  proportion  of  readers  who 
were  19-  to  23-year-old  males.  Five  magazines  were  selected:  Motor  Trend,  Motor¬ 
cyclist,  Cycle  World,  Car  Craft,  and  Popular  Hot  Rodding.  Information  obtained  from  the 
publishers  indicated  that  between  25  and  50  percent  of  the  subscribers  to  these  magazines 
were  persons  between  18  and  24  years  old  (the  closest  breakout  to  the  19-  to  23-year-old 
age  group  that  could  be  obtained).  Because  of  the  lack  of  precision  in  these  figures,  it 
was  difficult  to  determine  the  number  of  individuals  on  the  mailing  lists  who  were  in  the 
target  population.  In  addition,  female  subscribers  were  not  screened  from  the  lists;  it  was 


3 The  direct-mail  test  was  part  of  a  larger  experiment  designed  to  evaluate  the 
effectiveness  of  two  marketing  strategies,  singly  and  in  combination,  in  recruiting  a 
somewhat  older  population.  The  second  strategy  was  peer  networking  based  on  expanded 
use  of  the  Navy's  Recruiting  Assistance  Program.  The  test  of  this  strategy  was  cancelled, 
however,  because  of  problems  encountered  during  implementation  of  the  field  test,  and 
the  experiment  was  reduced  to  an  evaluation  of  only  the  direct-mail  strategy.  (See 
Fernandes,  K.  Evaluation  of  the  Navy's  Recruiting  Assistance  Program  as  a t  peer  net¬ 
working  strategy  for  recruiting  the  19-  to  23-year-old  market  (NPRuC  SR  83-11).  San 
Diego:  Navy  Personnel  Research  and  Development  Center,  January  1983.) 
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assumed  that  their  number  would  be  small  and  that  those  who  received  the  materials,  if 
not  interested  in  the  Navy,  might  pass  the  letter  on  to  a  male  friend  or  relative  who  was. 
Finally,  the  lists  were  not  screened  to  eliminate  individuals  who  were  outside  the  19-  to 
23-year-old  group  because  the  options  available  for  removing  them  were  either  prohibi¬ 
tively  expensive  or  of  questionable  reliability. 

Because  the  mailing  lists  contained  the  names  of  institutions  and  military  personnel, 
as  well  as  duplicate  subscribers,  the  following  adjustments  were  made  to  each  list: 

1.  For  the  experimental  group,  the  names  of  institutions  (e.g.,  schools,  businesses, 
and  libraries)  and  military  personnel  were  identified  and  eliminated  from  the  mailing  lists 
to  save  mailing  costs  and  obtain  a  more  realistic  base  for  the  calculation  of  response 
rates.  This  process  was  expensive  and  time-consuming;  consequently,  it  was  not  repeated 
for  the  control  group  where  there  were  no  mailing  costs  involved.  Instead,  adjustments 
were  made  by  applying  the  percentage  actually  eliminated  from  each  experimental  group 
list  to  the  number  of  names  on  the  corresponding  control  list.  The  resulting  figur  were 
used  when  calculating  control  group  response  rates.  Table  1  shows  the  number  c  tames 
that  remained  on  the  experimental  group  mailing  lists  after  institutional  and  litary 
names  were  removed  and  the  estimated  sizes  of  the  control  lists  if  the  same  p,  Mure 
had  been  used. 


Table  1 

Size  of  Experimental  Group  After  Removal  of  Institutional/ 
Military  Names  and  Estimated  Size  of  Control  Group 


Mailing  List 

% 

Removed3 

Number  Remaining 
in  Exp.  Group 

Estimated  Size  of 
Control  Group 

Popular  Hot  Rodding 

3.6 

1,949 

1,890 

Cycle  World 

5.6 

7,810 

7,180 

Motor  Trend 

4. 9 

13,362 

13,588 

Motorcyclist 

2.0 

3,601 

3,374 

Car  Craft 

3.5 

5,386 

4,699 

Total 

32,108 

30,731 

^he  percentage  actually  removed  from  experimental  group  mailing  lists  was  applied  to 
the  corresponding  control  group  list  to  adjust  the  size  of  the  group. 


2.  Although  some  names  appeared  on  several  mailing  lists  because  of  multiple 
subscriptions,  duplicates  were  not  eliminated  because  of  the  time  and  expense  required. 
Instead,  a  sample  of  names  drawn  from  the  experimental  and  control  mailing  lists  was 
checked  for  frequency  of  duplicates.  Based  on  this  procedure,  it  was  estimated  that  the 
total  number  of  names  would  be  reduced  by  6  percent  if  duplicates  were  removed.  This 
percentage  was  used  to  estimate  the  total  number  of  persons  in  the  experimental  and 
control  groups  as  a  whole. 

Field  Test  Procedures 


Direct  mail  materials  were  distributed  to  about  32,000  subscribers  in  two  waves— in 
February  1981  to  subscribers  to  Popular  Hot  Rodding  and  Cycle  World  and  in  May  1981  to 
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subscribers  of  the  other  three  magazines.  The  direct-mail  materials  consisted  of  the 
"school"  letter  (see  appendix)  and  two  postage-paid  reply  cards,  which  were  color-coded  to 
identify  the  mailing  list  on  which  the  name  appeared.  One  card  was  intended  for  the 
person  receiving  the  letter,  and  the  other  was  to  be  passed  on  to  a  friend.  Individuals 
interested  in  joining  the  Navy  were  requested  to  provide  their  name,  address,  telephone 
number,  date  of  birth,  and  educational  data  and  to  mail  the  card  to  a  west  coast  post 
office  box. 

The  reply  cards  were  screened  to  identify  individuals  who  were  16-  to  30-years  old  or 
who  did  not  provide  age  information.  These  individuals  were  considered  to  be  leads  (i.e., 
interested  in  the  Navy  and  appropriate  for  recruiters  to  follow  up  regarding  enlistment)'' 
and  their  names  and  addresses  were  forwarded  to  the  appropriate  recruiting  station.  Each 
month,  the  recruiters  provided  the  status  of  leads,  indicating  whether  a  contact  had  been 
attempted,  a  contact  had  been  made,  an  appointment  had  been  made,  or  an  interview  had 
been  conducted.  A  copy  of  the  lead  status  report  is  included  in  the  appendix.  If  no  action 
had  been  taken  on  a  particular  lead,  recruiters  gave  the  reason  and  the  lead  was  included 
in  the  next  status  report. 

Recruiting  stations  were  contacted  by  telephone  when  status  reports  were  not 
returned  or  when  misunderstandings  about  procedures  developed.  Although  the  follow-up 
system  provided  information  on  the  initial  action  taken  on  each  lead,  it  did  not  indicate 
the  extent  of  the  contact  before  the  lead  lost  interest,  was  found  to  be  ineligible  for 
military  service,  or  finally  enlisted. 

Evaluation 

Impact  Assessment 

The  following  measures  were  used  to  evaluate  the  impact  of  the  direct  campaign. 

1.  The  rate  of  return  was  determined  by  dividing  the  number  of  reply  cards 
received  by  the  total  number  of  names  for  the  experimental  group  as  a  whole.  The  rate  of 
eligible  returns  was  determined  by  dividing  the  number  of  eligible  leads  (i.e.,  leads  who 
were  17-to-30  years  old  and,  therefore,  eligible  to  enlist  during  the  field  test  period)  by 
the  number  of  names  on  a  mailing  list.  The  rate  of  eligible  returns  was  calculated  for 
each  experimental  list. 

2.  The  conversion  rate  was  determined  by  dividing  the  number  of  enlistments  by  the 
number  of  eligible  leads.  A  single  conversion  rate  was  calculated  for  the  experimental 
group  as  a  whole. 

3.  The  enlistment  rate  for  each  experimental  and  control  mailing  list  was  deter¬ 
mined  by  dividing  the  number  of  enlistments  by  the  number  of  names  on  a  list.  The 
adjusted  enlistment  rate,  which  was  calculated  for  the  experimental  or  control  group  as  a 
whole,  was  determined  by  dividing  the  number  of  enlistments  for  the  group  by  the 
estimated  number  of  persons  in  the  group  (i.e.,  the  number  of  names  in  the  group  reduced 
by  the  estimated  percentage  of  duplicate  names). 


"Individuals  who  were  16  years  old  were  considered  to  be  valuable  contacts  for  future 
enlistment  and  so  were  considered  to  be  leads  even  though  they  could  not  be  enlisted 
during  the  field  test  period.  Individuals  who  were  31-  to  34-years  old,  although  eligible  to 
enlist,  were  considered  to  be  marginal  for  follow-up  and  enlistment  and  so  were  not 
included  as  leads. 


Enlistment  records,  consisting  of  name,  age,  gender,  and  recruiting  station  where 
enlisted,  were  extracted  from  the  Personalized  Recruiting  for  Immediate  and  Delayed 
Enlistment  (PRIDE)  file  for  NRDs  San  Francisco  and  Seattle  for  the  period  from  March 
through  October  1981.  These  records  were  compared  with  the  mailing  lists  consisting  of 
subscriber  name,  address,  and  zip  code  to  determine  whether  or  not  a  subscriber  had 
enlisted.  Since  only  the  name  and  zip  code  were  common  to  the  enlistment  records  and 
mailing  lists,  the  following  criteria  were  used  to  identify  matching  entries: 

l.  The  enlistment  record  and  the  mailing  list  entries  had  to  have  the  same  last 
name. 


2.  The  recruiting  stations  for  the  two  entries  had  to  match;  that  is,  the  zip  code  of 
the  mailing  list  entry  had  to  be  in  the  territory  of  the  recruiting  station  of  the  enlistment 
record  entry. 

3.  The  information  on  the  enlistment  record  and  the  mailing  list  entries  had  to 
match;  for  example,  agreement  between  the  first  name  and/or  first  initial  and  no 
difference  in  gender  such  as  "Mrs."  on  one  and  "M"  on  the  other. 

It  was  assumed  that  any  errors  associated  with  the  application  of  these  criteria  would 
be  the  same  for  both  the  experimental  and  control  groups.  For  example,  because  these 
criteria  excluded  persons  who  enlisted  at  any  recruiting  station  other  than  the  one 
assigned  to  the  mailing  list  zip  code,  the  number  of  enlistments  and  hence  the  conversion 
and  enlistment  rates  obtained  were  likely  to  be  underestimated  for  both  the  experimental 
and  control  groups. 

Qualitative  Analyses 

The  age  and  educational  characteristics  of  the  leads  and  enlistees  in  the  experimental 
and  control  groups  and  the  two  NRDs  were  compared  to  determine  if  the  direct-mail 
materials  had  attracted  the  same  type  of  person,  in  terms  of  these  demographic 
characteristics,  as  had  enlisted  during  the  field  test  period.5  In  addition,  the  pass-on  rate, 
defined  as  the  percentage  of  leads  not  on  the  experimental  group  mailing  lists,  was 
calculated  for  the  leads  received.  Although  the  direct-mail  materials  included  special 
pass-on  cards,  the  pass-on  rate  was  not  calculated  directly  from  these  cards  because  of 
the  possibility  that  an  incorrect  card  had  been  returned. 

Eligible  leads  were  interviewed  by  telephone  to  determine  their  reactions  to  the 
direct-mail  letter  and  to  contacts  with  a  recruiter.  These  semistructured  and  open-ended 
interviews  were  conducted  in  August  1981  using  the  interview  guide  presented  in  the 
appendix.  Telephone  contact  was  attempted  with  every  eligible  lead  whose  name  had 
been  forwarded  to  a  recruiting  station;  55  interviews  were  completed. 

The  monthly  status  reports  were  examined  to  determine  the  extent  to  which 
recruiters  had  attempted  contact  with  the  leads  provided  them. 


*Gender  information  was  not  requested  on  the  reply  cards  and  could  not  be 
accurately  estimated  from  the  names  of  the  leads.  Hence,  gender  was  not  included  in  the 
comparison  of  demographics. 
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RESULTS 

Effectivcjyss  of  the  Direct-mail  Campaign 

The  return  rate  of  the  direct-mail  campaign  was  1.14  percent.  Of  the  367  cards 
received,  177  met  the  age  criterion  (i.e.,  16-  to  30-years  old)  to  be  considered  a  lead  and 
were  forwarded  to  the  recruiting  stations  f  ^  follow-up.  The  overall  rate  of  eligible 
returns  was  .55  percent.  Table  2  presents  the  frequency  and  rate  of  eligible  returns  for 
each  mailing  list. 


Table  2 

Frequency  and  Rate  of  Eligible  Returns 
by  Mailing  List 


Mailing  List 

Number  of 

Eligible  Returns 

Return  Rate 
(%) 

Popular  Hot  Rodding 

19 

0.97 

Cycle  World 

62 

0.79 

Motor  Trend 

45 

0.34 

Motorcyclist 

24 

0.67 

Car  Craft 

27 

0.50 

Total 

177 

0.55 

Two  of  the  leads  enlisted.  The  conversion  rate  was  1.29  percent,  based  on  the  155 
leads  who  were  17-  to  30-years  old  and,  therefore,  eligible  to  enlist.  Given  the  criteria 
used  to  match  enlistment  record  and  mailing  list  entries,  this  figure  probably  under¬ 
estimated  the  true  conversion  rate  for  the  experimental  group.  The  rate  obtained  was 
lower  than,  but  not  significantly  different  from,  the  COMNAVCRUITCOM  conversion  rate 
of  2.65  percent  for  national  leads  in  Navy  Recruiting  Area  Eight  in  FY81.  The  following 
characteristics  of  national  leads  should  be  considered  when  comparing  the  two  conversion 
rates: 


1.  National  leads  include  responses  to  both  print  and  direct-mail  advertising. 

2.  National  leads  are  screened  for  both  education  and  age  and  include  individuals  up 
to  34  years  of  age. 

3.  The  national  lead  conversion  rate  is  calculated  from  information  generated  from 
follow-up  surveys  with  recruiting  stations. 

Because  of  these  differences,  it  was  not  possible  to  determine  the  extent  to  which  the 
Area  Eight  conversion  rate  was  an  overestimate  relative  to  the  rate  for  the  experimental 
group. 

A  comparison  of  the  adjusted  enlistment  rates  (i.e.,  the  enlistment  rates  based  on  the 
number  of  persons  in  each  group)  for  the  experimental  and  control  groups  indicated  that 
the  rates  were  not  significantly  different.  As  shown  in  Table  3,  the  adjusted  enlistment 


6 


Table  3 


Number  of  Enlistments  and  Enlistment 
Rates  by  Mailing  List 


Mailing  List 

Experimental  Group 

Control  Group 

Number 

Rate  (96) 

Number 

Rate(96) 

Popular  Hot  Rodding 

4 

0.21 

7 

0.37 

Cycle  World 

15 

0.19 

15 

0.21 

Motor  Trend 

21 

0.16 

21 

0.15 

Motorcyclist 

7 

0.19 

12 

0.36 

Car  Craft 

13 

0.24 

15 

0.32 

Total 

59 

0.20a 

64 

0.22a 

^he  total  rates  are  based  on  the  number  of  persons  estimated  to  be  in  the  experimental 
and  control  groups  if  duplicate  names  were  removed. 


rate  for  the  experimental  group  was  .20  percent,  compared  to  .22  percent  for  the  control 
group.  The  enlistment  rates  (i.e.,  the  enlistment  rates  based  on  the  number  of  names)  for 
the  individual  lists  are  also  shown  in  Table  3.®  The  enlistment  rate  for  the  experimental 
group  was  lower  than  that  for  the  control  group  for  four  of  the  five  magazines. 

Qualitative  Analyses 

Table  4  presents  the  age  and  education  of  the  leads  generated  in  the  field  test  and  of 
the  enlistees  in  the  experimental  and  control  groups  and  in  the  two  NRDs  during  the 
period  of  the  field  test.  Twenty-four  percent  of  the  leads  were  in  the  19-  to  23-year-old 
group.  This  figure  was  at  the  low  end  of  the  percentage  range  of  subscribers  the 
magazines  estimated  to  be  in  the  target  age  group  and  lower  than  the  percentage  of  19- 
to  23-year-old  enlistees  in  the  experimental  and  control  groups  and  in  the  two  NRDs. 
The  percentage  of  leads  who  had  completed  at  least  12  years  of  education  was  similar  to 
that  for  enlistees  in  the  experimental  and  control  groups  and  in  the  two  NRDs;  however,  a 
substantially  higher  percentage  of  the  leads  had  completed  some  college. 

In  terms  of  pass-on  rate,  44  percent  of  the  leads  did  not  appear  on  any  of  the  mailing 
lists.  Because  the  names  and  addresses  of  these  leads  differed  from  those  of  the 
subscribers,  they  were  considered  to  be  true  pass-ons.  Another  16  percent  of  the  leads 
had  the  same  last  name  and  address,  but  a  different  first  name,  as  an  entry  on  the  mailing 
lists.  In  these  cases,  the  identification  as  a  pass-on  was  questionable  since  the  person  who 
returned  the  card  may  have  been  the  actual  magazine  reader,  while  the  person  who  was 
listed  paid  for  the  subscription. 


‘The  procedure  used  to  estimate  duplicates  was  appropriate  to  apply  to  the 
experimental  and  control  groups  as  a  whole  but  not  to  the  individual  mailing  lists. 


7 


Table  4 


Age  and  Education  of  Leads  Generated  in  the  Field  Test  and  Enlistments 
in  the  Experimental  and  Control  Groups  and  in  the  Two  NRDs 


Characteristic 

Direct-mail  Leads 
(%)  (N  =  177) 

Exp.  Group 
(%)  (N  =  59) 

Enlistment 

Cont.  Group 
(%)  (N  =  64) 

NRD  (%) 
(N  =  5909) 

Age 

16 

12.4 

— 

— 

— 

17 

14.7 

18.6 

14.1 

17.8 

18 

11.9 

27.1 

29.7 

24.7 

19-23 

24.2 

40.7 

51.6 

40.6 

24  and  older 

24.9 

13.6 

4.7 

16.8 

Not  available 

11.9 

— 

— 

0.1 

Education  (Highest  grade) 

9  and  below 

4.0 

0.0 

1.6 

3.3 

10 

16.9 

10.2 

14.1 

9.8 

11 

16.9 

28.8 

18.8 

26.2 

12 

37.3 

54.2 

59.4 

50.2 

13  and  above 

20.3 

1.7 

4.7 

6.4 

Not  available 

4.5 

5.1 

1.6 

4.1 

Of  the  leads  who  participated  in  the  telephone  interviews,  22  percent  were  in  the  19- 
to  23-year-old  age  group  and  68  percent  had  a  high  school  diploma.  Because  the 
interviewees  were  not  selected  randomly,  the  survey  findings  may  not  be  representative 
of  the  group  as  a  whole.  Given  this  qualification,  the  following  summarizes  the  interview 
responses: 

1.  About  two-thirds  of  the  interviewees  recalled  receiving  a  direct-mail  letter  from 
the  Navy,  but  most  did  not  remember  its  contents. 

2.  About  one-third  of  the  interviewees  indicated  that  they  had  visited  a  recruiter. 
All  of  them  had  positive  impressions  of  the  recruiter,  whom  they  perceived  as 
informative,  t’uthful,  and  professional. 

3.  Over  half  of  the  interviewees  (and  over  three-fourths  of  those  who  saw  a 
recruiter)  indicated  that  they  would  consider  future  service  in  the  Navy.  The  primary 
reason  for  considering  enlistment  was  educational  advancement  and  skill  development. 
The  primary  objection  to  Navy  service  was  a  better  alternative  such  as  a  good  job  or 
career,  or  plans  to  attend  college. 

Although  recruiters  frequently  delayed  their  follow-up,  they  attempted  to  contact 
(e.g.,  locate  a  telephone  number,  mail  out  recruiting  material,  set  up  an  interview)  90 
percent  of  the  leads  sent  to  the  recruiting  stations.  An  examination  of  the  lead  status 
reports  (supplemented  with  the  telephone  interview  information)  indicated  that  one- 
fourth  of  the  leads  turned  out  to  be  either  invalid  or  ineligible  to  enlist  (excluding  the 
leads  known  to  be  16  years  of  age  who  had  been  forwarded  for  follow-up).  Invalid  leads 
included  pranks  and  individuals  who  had  changed  residences  and  could  not  be  contacted. 
Ineligible  leads  included  individuals  who  could  not  enlist  because  of  physical  disability, 
mental  aptitude,  education,  or  police  record. 
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DISCUSSION 


A  number  of  factors  should  be  considered  in  interpreting  the  lack  of  impact  of  the 
direct-mail  campaign  on  enlistments.  First,  the  size  of  the  campaign  and  the  number  of 
subscribers  in  the  target  population  may  have  been  insufficient  to  provide  a  fair  test  of 
the  strategy.  The  selection  of  recruiting  stations  was  limited  to  a  subset  of  those  in 
NRDs  San  Francisco  and  Seattle  because  of  other  ongoing  recruiting  research  and  the 
yoking  of  the  direct-mail  strategy  with  a  second  strategy  in  the  initial  experimental 
design.  There  may  not  have  been  enough  magazine  subscribers  in  the  territories 
surrounding  the  NRD  stations  to  produce  stable  estimates  of  enlistment  and  conversion 
rates  for  evaluating  the  effectiveness  of  the  campaign.  Furthermore,  although  the 
magazines  were  selected  for  their  high  proportion  of  19-  to  23-year-old  subscribers,  the 
desired  targeting  of  the  campaign  was  reduced  to  the  extent  that  the  letter  was  received 
by  individuals  outside  this  age  group.  In  future  efforts  of  this  type,  the  availability  of 
specialized  mailing  lists  that  contain  large  numbers  of  target  group  members  needs  to  be 
explored . 

Second,  it  is  possible  that  no  matter  what  the  direct-mail  market  approach,  it  would 
have  had  only  a  small  impact  on  enlistments.  For  example,  if  the  field  test  had  been  as 
effective  as  COMNAVCRUITCOM's  direct-mail  campaigns  and  had  resulted  in  a  conver¬ 
sion  rate  of  2.65  percent,  the  current  effort  would  have  resulted  in  4  (rather  than  2)  of 
the  155  eligible  leads  enlisting.  Direct  mail  campaigns  may  only  have  a  significant  effect 
on  conversion  rates  when  they  are  combined  with  other  strategies  such  as  peer  net¬ 
working  as  originally  planned. 

The  qualitative  analyses  indicated  that  the  direct-mail  letter  may  not  have  appealed 
to  members  of  the  target  population.  The  leads  generated  by  the  campaign  underrepre¬ 
sented  the  19-  to  23-year-old  age  range  in  comparison  to  the  percentage  of  subscribers 
estimated  by  the  magazines  to  be  in  the  target  group  and  in  comparison  to  the  age 
distribution  of  enlistees  in  the  two  NRDs  during  the  field  test  period.  Furthermore,  the 
content  of  the  letter  was  not  "memorable"  enough  to  be  recalled  by  most  of  the  leads 
interviewed  in  the  follow-up  telephone  survey.  Although  the  letter  was  evaluated 
positively  by  members  of  the  target  population  during  copytesting,  the  results  indicated 
that  it  may  be  difficult  to  tailor  a  recruiting  message  to  appeal  to  a  specific  age  group. 

The  lack  of  impact  of  the  direct-mail  campaign  should  be  considered  in  light  of  the 
recruiting  environment  in  which  the  field  test  took  place.  During  the  field  test,  both  of 
the  participating  NRDs  were  usually  attaining  or  exceeding  their  monthly  recruiting  goals. 
In  such  an  environment,  recruiters  may  have  given  direct-mail  leads  a  lower  priority  for 
follow-up  than  leads  from  other  sources  that  they  considered  more  promising  to  produce 
enlistments. 

Although  there  were  problems  associated  with  the  field  test  design,  several  of  the 
measures  calculated  may  be  applicable  to  other  direct-mail  evaluations.  These  measures 
are  the  enlistment  rate  and  the  conversion  rate,  both  generated  by  matching  names  on  the 
mailing  lists  with  enlistment  records.  These  measures  need  to  be  compared  with  the 
conversion  measure  used  by  COMNAVCRUITCOM  that  involves  follow-up  surveys  to 
recruiting  stations  to  determine  the  percentage  of  leads  converted  into  enlistments.  Such 
a  comparison  would  provide  information  on  the  validity  of  both  measurement  techniques 
and  might  enhance  the  accuracy  of  the  methods  used  to  evaluate  direct-mail  efforts. 


CONCLUSIONS 

The  direct-mail  campaign  did  not  have  a  significant  impact  upon  enlistment  rates  and 
was  not  effective  in  appealing  to  the  target  population  of  19-  to  23-year-olds. 
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APPENDIX 

MATERIALS  USED  IN  FIELD  TEST  AND  EVALUATION 


Direct-mail  Letter . 

Sample  Lead  Status  Report .  .  . 
Telephone  Survey  Interview  Guide 


A-0 


IN  REPLY  REFER  TO 

CNRC/22  3/ j  f 
1100 

30  April  1981 


Dear  Friend, 

Would  you  be  willing  to  invest  a  few  years  to  get  control 
over  your  life? 

Training  and  experience  are  the  answers.  But  where  do  you 
get  them?  Most  companies  don't  hire  people  without  experience. 

Even  recent  graduates  of  technical  schools  don't  have  experience. 

There  is  a  "school"  that  offers  both  training  and  experience. 
It's  the  U.  S.  Navy.  One  of  the  biggest  users  of  the  most  modern 
equipment  and  machinery  in  the  world. 

Navy  training  is  among  the  best.  You  may  work  on  equipment 
from  the  big  companies.  Companies  looking  for  experienced  people. 
You  couldn't  pay  for  this  type  of  learning. 

But  the  Navy  is  one  "school"  that  pays  you.  Wages  a  lot 
higher  than  other  schools.  Benefits  like  food  and  housing,  health 
care  and  insurance,  plus  30  days  vacation  earned  every  year. 

And  the  Navy  takes  care  of  its  own.  It’s  a  place  where  your 
workmates  will  probably  be  your  best  friends.  And  your  supervisors 
will  be  there  to  show  you  the  ropes  and  help  you  move  ahead  when 
you're  ready. 

To  learn  more,  send  me  back  the  prepaid  postcard.  There's 
no  obligation. 

And  the  next  time  you  lose  out  on  a  job,  ask  the  guy  who  got 
it.  He  just  may  be  former  Navy. 

Sincerely, 

R.  N.  THEOFSL  \ 

Lieutenant,  u.  S.  Navy 
Head,  Field  Liaison  Branch 


FOR  EACH  OF  THE  RETAIN  LEADS  LISTED  BELOW,  EITHER:  *Still  pending  from 

(1)  WRITE  IN  THE  REASON  IF  NO  ACTION  HAS  BEEN  TAKEN  YET,  OR  a  former  report 
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"HASt  FOLD  THIS  FORM  ONCE  WITH  WESTAT'S  ADDRESS  SHOWING,  STAPLF ,  AND  PUT  IN  MAILBOX.  THANK  YOU. 


WEST  AT  DIRECT-MAIL 
EVALUATION  TOPIC  GUIDE 
CONTRACT  N00123-79-C-1511 


Hello,  my  name  is _ .  I'm  with  Westat,  an  independent  survey 

research  firm  in  Rockville,  Maryland.  Westat  is  doing  research  for  the  U.S.  Navy  on  their 
advertising  campaigns.  We're  conducting  interviews  with  men  and  women  like  yourself  on 
the  West  Coast  and  would  like  to  include  your  ideas. 

We  selected  your  name  randomly  from  subscriber  mailing  lists  of  popular  magazines. 
The  information  you  provide  us  will  remain  confidential  and  will  not  be  given  to  anyone  in 
the  military.  Your  participation  is  totally  voluntary. 

Do  you  have  any  questions? 
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Find  out  whether  respondent  received  a  letter  in  the  mail  from  the  Navy  in  past  few 
months  telling  about  a  job  opportunity.  Probe: 

•  Whether  they  read  it 

•  What  they  remember  about  it 

2.  Determine  whether  information  in  letter  affected  their  feelings  about  the  Navy. 
Probe: 

•  Degree  to  which  it  got  them  interested  in  finding  out  more  about  Navy,  and 
why  (or  why  not) 

•  Degree  to  which  it  made  them  wish  to  see  a  Navy  recruiter,  and  why  (or  why 
not) 

3.  Find  out  if  they  did  anything  to  get  more  information  about  Navy.  If  saw  Navy 
recruiter,  find  out  what  led  up  to  their  contact. 

4.  (IF  SAW  RECRUITER)  Discuss  what  happened  when  they  saw  recruiter.  Probe: 

•  Type  of  person  recruiter  was 

•  Things  recruiter  said  and  did  which  made  them  feel  more  positive  about 
Navy 

•  Things  said  and  done  which  made  them  feel  more  negative  about  Navy 

•  Things  recruiter  could  have  said  and  done  (realistic)  which  would  have  made 
them  feel  more  like  joining  the  Navy 

5.  Probe  them  to  find  out  whether  they  will  consider  service  in  the  Navy  for  the 
future.  Reasons. 

6.  Obtain  information  on: 

•  Age 

•  Marital  status 

•  Education  completed 

•  Prior  military  service 

•  Sex  (DO  NOT  ASK) 
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DISTRIBUTION 


Chief  of  Naval  Operations  (OP- 11 5),  (OP- 135) 

Chief  of  Naval  Education  and  Training  (02),  (N-5) 

Commander,  Navy  Recruiting  Command  (Code  223) 
Commander,  Navy  Recruiting  Area  Eight 
Commanding  Officer,  Navy  Recruiting  District,  San  Francisco 
Commanding  Officer,  Navy  Recruiting  District,  Seattle 
Defense  Technical  Information  Center  (DDA)  (12) 


